feat(experimental): add WAN ReFL as a self-contained training package by YSunLIN · Pull Request #210 · Tencent-Hunyuan/UniRL

YSunLIN · 2026-07-15T07:47:49Z

feat(experimental): add WAN ReFL as a self-contained training package

Summary

This PR moves algorithm-specific training flows out of UniRL core and organizes them as self-contained recipes, landing experimental/refl — WAN ReFL/BPTT (direct differentiable-reward backprop) for Wan 2.1 T2V and Wan 2.2 I2V — as the first recipe. The recipe is built purely on existing core seams (placement() + remote_hydra + _target_ late binding); it introduces no new orchestration layer.

experimental/refl package (trainer + roles + models + reward + examples + README, self-contained):
- REFLTrainer subclasses unirl.trainer.base.BaseTrainer directly and wires two colocated roles (actor + differentiable reward), mirroring RewardBackpropTrainer (the SD3 image-ReFL driver).
- ReflActorRole mirrors ReFLPolicy's family-agnostic contract: pipeline_target + model_config → Pipeline.from_config, with FSDPBackend composed in initialize(). Payloads are sample-native primitives (Texts / Images + per-sample metadata records) — no request objects.
- Recipe-local Wan21ReflPipeline / Wan22ReflPipeline subclass the mainline pipelines and swap in a diffuse_with_grad BPTT stage (truncated mid_timestep..final_timestep grad window, single-branch CFG, optional per-step KL against the LoRA-disabled reference). The BPTT stage contract (DiffuseWithGradResult) is recipe-local by design: concrete stages inherit the DiffusionStage Protocol explicitly, so a protocol-level stub would become a real None-returning method on every stage and defeat hasattr capability checks; the promotion path is a separate opt-in protocol (the DifferentiableReward idiom) once a second out-of-recipe BPTT consumer exists. Mainline WAN diffusion/pipeline code is untouched.
- Recipe-local rewards with isolated dependencies: VideoAlign (Qwen2-VL VQ/MQ/TA) and Face-identity scorers under experimental/refl/reward/, each with its own requirements.txt (core env stays clean).
- Flat configs (same schema as examples/): experimental/refl/examples/{wan21_t2v_videoalign_refl,wan22_i2v_face_refl}.yaml.
Core enablers (framework-level, shared with future BPTT recipes):
- Differentiable path: cross-RPC grad-input handling fix in Worker.call; RewardService.score_differentiable(media_tensor, prompts, records) widened to video tensors + per-sample metadata; DifferentiableReward protocol updated to match.
- Memory-optimized WanVideoVAE (nested grad checkpoint + activation-grad-only conv) deliberately replaces the WAN decode implementation in core — memory/numeric VAE optimizations are model assets shared by GRPO and ReFL, not algorithm property; forking the decode contract per algorithm was rejected in review. Hub repo-id checkpoints (the mainline wan21_t2v.yaml default) resolve through the HF cache (snapshot_download), preserving the previous from_pretrained loading semantics.
- LoraConfig.module_prefix (Wan 2.2 trains only the low_noise DiT), linear_warmup LR schedule, MultimodalRLDataSource run.shuffle.

Review adjustments (2026-07-30, maintainer push at `ec561c7a`)

Fast-forward on top of the author's head e3c6b940 (nothing rewritten), applying the review consensus and re-aligning with main:

Rebased onto the sample-native core: upstream feat(agentic): add Sample-native multi-turn rollout and training #214 removed RolloutReq/RolloutInputs after this PR's last merge; the recipe now consumes Sample primitives end-to-end.
Dropped the roles DSL (recipes/common/, −422 lines) per review: REFLTrainer(BaseTrainer) + remote_hydra replace the role-list orchestration; configs re-rooted from roles: to the repo-wide flat schema. Net −322 lines vs the previous head while keeping all functionality.
P1 KL DP-aggregation fix: kl_loss was a per-shard scalar shared_field — DP collect kept only rank 0's KL and re-broadcast it, which only lined up on the verified batch == actor_dp == 8 topology (logs duplicated rank 0; other B/dp splits risked shape mismatch). Now a per-sample [B] concat column; scripts/verify_refl_kl_batching.py pins the wire-layer invariants across B==dp / B>dp / non-power-of-two / dp==1 / unequal actor-reward dp (standalone script per the test: remove tests directory #99/test: remove tests directory #267 no-unenforced-test-tree policy).
Seed scheme: kept your fixed-noise semantics (params.seed used verbatim, every rollout/rank — the regime your 835-rollout curve was trained in). An earlier adjustment decorrelated noise per rollout/rank; a checking pass flagged that as an unevidenced semantics change and reverted it (fe2788c6).
I2V condition assembly moved from the role into Wan2xReflPipeline.build_refl_conditions (mirrors each mainline generate); the role has no per-model imports.

Expected behavior deltas vs the curves below: per-shard-correct KL values (the Wan 2.2 kl_weight=1.0 loss/KL curves may shift slightly — that is the correction) and the bf16 LoRA master (fp32 master trips a torch 2.7 FSDP2 uniform-dtype assert; a dedicated fp32 LoRA param-group fix is planned as a follow-up). Curve shapes are otherwise expected to match.

Second pass (e0497399), closing the remaining review findings:

Dependency contract: colocated reward+actor share one Python process, so a recipe requirements.txt can only add packages — it can never version-"isolate". The old file pinned transformers==4.45.2 / peft==0.10.0 / flash-attn==2.5.8, which would downgrade the core stack (transformers>=5.6, peft>=0.14) on install. VideoAlign now runs on the core stack: the wrapper's documented 5.x shim is enabled (explicit mm_token_type_ids black-list pop — never signature filtering, which PEFT breaks), flash-attn falls back to SDPA when absent, and requirements.txt is additive-only.
Hub VAE loading: WanVideoVAE.load_from_diffusers reads local files only; WAN21Bundle now resolves non-local paths through the HF cache first, so mainline repo-id configs keep loading.
Recipe-local BPTT contract: DiffuseWithGradResult + the diffuse_with_grad contract moved out of the core DiffusionStage Protocol (explicitly-inherited protocol stubs made hasattr capability checks vacuously true on every stage) into experimental/refl/models/types.py.
Single KL knob: actor.kl_weight owns on/off + weight and is injected into the stage's sampler_kwargs; a stale sampling.sampler_kwargs.kl_weight raises instead of being silently overridden.
Hygiene: BPTT window validated (0 <= mid <= final < T) at diffuse time; launch scripts run under set -euo pipefail; the Face reference-embedding cache is a bounded LRU (64).
Layer + naming decisions (maintainer review): the top-level layer is experimental/ — core carries the official training flows, and a layer named recipes read as the official usage (the word also already means the flat YAML here). experimental/ names the incubation side of the intended two-way flow: mainstream packages get absorbed and solidified into core; internal/private packages live here uncommitted. Inside a package, directories are named after the core home their content graduates into (structural no-op promotion): examples/ mirrors the top-level examples/, reward/ mirrors unirl/reward/, models/ mirrors unirl/models/ (the BPTT pipeline/stage subclasses graduate by merging into the matching model packages). Launch .sh wrappers were deleted — the launch surface is one documented command, now in experimental/refl/README.md together with the layout map, environment policy, and the verification table.

Test Plan

8xH20 with a Ray cluster started (RAY_ADDRESS=auto); replace /path/to/... with real paths.

export DATA_PATH=/path/to/prompts.txt PRETRAINED_MODEL=/path/to/Wan2.1-T2V-1.3B-Diffusers \
       VIDEOALIGN_MODEL_PATH=/path/to/VideoReward
RAY_ADDRESS=auto python -m experimental.refl.run --config-name=wan21_t2v_videoalign_refl num_devices=8

pip install -r experimental/refl/reward/face/requirements.txt
export PRETRAINED_MODEL=/path/to/Wan2.2-I2V-A14B-Diffusers DATA_PATH=/path/to/i2v_prompts.jsonl \
       FACE_MODEL_PATH=/path/to/antelodev2
RAY_ADDRESS=auto python -m experimental.refl.run --config-name=wan22_i2v_face_refl num_devices=8

Static validation at e0497399 (CPU, 2026-07-30):

compose-check both configs (python -m experimental.refl.run --config-name=... --cfg job --resolve) — rc=0 / rc=0;
scripts/check_recipe_targets.py — 2270 _target_ paths resolve (incl. recipes.*);
worker-walker simulation instantiates every nested config dataclass and binds the ReflActorRole ctor for both configs;
python scripts/verify_refl_kl_batching.py — all topology checks pass;
full pre-commit run --all-files passes; unirl/ imports nothing from experimental/.

Maintainer GPU smoke (2026-07-30, head 40b3f4c9, 8xH20 fleet image unirl-v1d: torch 2.7.1 / transformers 4.56.0.dev0 / peft 0.17.1):

wan21_t2v_videoalign_refl (num_rollouts=2, full 81f/480x832 geometry, local WAN ckpt + VideoReward-2B): PASS, clean exit — rollout 1/2 reward=1.9839 loss=-0.4960 grad_norm=31.1 (230s), rollout 2/2 reward=3.1245 grad_norm=25.3 (228s). Finite nonzero grad_norm = gradients flow reward → VAE → DiT LoRA across the RPC chain.
Two portability fixes found by this smoke and pushed (d96800dc, 40b3f4c9): REFLTrainer now stores cfg (BaseTrainer does not retain it), and the configs drop master_dtype: fp32 — fp32 LoRA over the bf16 base trips torch 2.7 FSDP2's uniform-original-dtype assert (the documented refl_sd3.yaml note); LoRA master weights are now bf16, matching the SD3 refl recipe.

Still pending on GPU (owner: @YSunLIN):

wan22_i2v_face_refl smoke at the current head (needs the face env extras + your I2V dataset with ref_video_path metadata);
the ≥4.58/5.x mm_token_type_ids path of the VideoAlign shim (fleet image is 4.56 → the pop is a no-op there; needs a transformers>=4.58 env);
longer-run curve-shape comparison vs your original curves (expected deltas: per-shard-correct KL, bf16 LoRA master);
one mainline WAN sanity: a short official wan21_t2v trainside run (or Hub repo-id load + decode eyeball) covering the WanVideoVAE swap;
provenance: upstream source + revision + license for WanVideoVAE and the VideoAlign port, per the repo vendor/VENDOR_COMMIT.txt convention.

Verification Status

Config Name	Model	Hardware	Status	Notes
`wan21_t2v_videoalign_refl`	Wan2.1-T2V-1.3B + VideoAlign reward	8xH20	Verified (pre-adjustment head)	ReFL/BPTT; `num_rollouts=835`; `batch_size=8`; `num_inference_steps=25`; final-step differentiable path
`wan22_i2v_face_refl`	Wan2.2-I2V-A14B + Face reward	8xH20	Verified (pre-adjustment head)	ReFL/BPTT; `num_rollouts=690`; `batch_size=8`; `module_prefix=low_noise`; `num_inference_steps=8`

Wan 2.1 T2V ReFL training reward curve

Wan 2.2 I2V ReFL training reward curve

Compatibility / Risk

experimental/refl is a new, self-contained path; existing entrypoints and the mainline WAN GRPO flow are unaffected. experimental/ is repo-run only for now (not in wheel packaging) — a follow-up recipes-contract PR will settle packaging + import-direction CI.
RewardService.score_differentiable signature changed (media_tensor, prompts, records); the two core ReFL call sites (unirl/trainer/refl.py) are updated in this PR. A follow-up PR will remove the legacy core ReFL path (train_refl.py / trainer/refl.py / train/refl/) so refl lives only in recipes.
The WAN VAE decode flow is replaced by the memory-optimized WanVideoVAE; dtype/tiling/recompute may differ slightly from the old path numerically — monitored via reward curves and sample quality.
MultimodalRLDataSource.run.shuffle defaults to true (previous behavior preserved); module_prefix is opt-in and backward compatible — confirm trainable params via startup logs.
Recipe reward requirements.txt files are additive-only by contract (same-process colocation makes version isolation impossible); rewards that genuinely need a conflicting stack belong in the out-of-process unirl-reward-service, which is only an option for non-differentiable rewards.

AI-assisted: the 2026-07-30 adjustment was implemented and statically validated with Claude Code under maintainer direction (haonan3); every change reviewed against the review thread consensus. Duplicate-work check: supersedes the orchestration-layer portion only; overlaps were reconciled with #188's direction in the review discussion.

haonan3 · 2026-07-16T12:57:53Z

The tiled VAE decode path hardcodes the output buffer batch dimension to 1, while hidden_states may have B > 1. The decoded tile therefore has batch size B and cannot be accumulated in-place into values when B > 1.

Since decode() now defaults to tiled=True, this can break existing WAN recipes such as wan21_t2v.yaml, which uses forward_batch_size: 4. Please allocate these buffers using the actual input batch size, apply the same fix to tiled_parallel_decode, and add a regression test covering batch size greater than one.

celve · 2026-07-22T06:33:57Z

I support moving WAN ReFL algorithm code into recipes/, but I don't think the new Trainer + Role abstraction is justified. Remote, Handle, DevicePool, placement, remote_hydra, and BaseTrainer already provide role lifecycle, placement, dispatch, and resource management.

Currently there is only one Trainer subclass and one Role subclass, while the proposal adds roughly 590 lines and a second orchestration API. The recipe still hardcodes actor and reward, so the roles list does not make the algorithm genuinely generic. There are also design concerns: Role assumes a fixed bundle → pipeline → backend lifecycle, and RewardRole bypasses RewardService.__init__ while changing its method contract.

I suggest removing unirl/trainer/trainer.py and recipes/common/roles.py, having REFLTrainer inherit BaseTrainer, and making the recipe worker inherit Remote directly. Existing placement/remote_hydra APIs can build the topology, while RewardService can be extended for differentiable image/video scoring with metadata. The WAN ReFL functionality should remain; the generic role DSL can be reconsidered once multiple independent recipes demonstrate the need.

- Remove trainer-side config validation and simplify mean calculation with np.mean. - Remove unused fit=train handling from trainer. - Rename REFL recipe namespace from refl_wan to refl. - Rename base_role.py to role.py for the REFL role implementation. - Move max_grad_norm handling out of the base role and let reward backends own it. - Add a generic reward role under unirl/reward that directly extends the reward service. - Remove reward package pip installation from pyproject configuration. - Remove hardcoded local paths from recipe configs and shell scripts. - Add requirements.txt files for face and videoalign reward backends.

… the wan refl recipe.

…atch size to 1

- ReflActorRole inherits Remote directly: inline initialize (build bundle/pipeline/backend) + step/save_checkpoint/load_checkpoint, and move RoleStepResult here; delete recipes/common/roles.py - reward role now uses unirl.reward.service.RewardService directly; Trainer.create_remote_role special-cases it so the worker builds the backend, other roles keep the cfg-driven path - RewardService.score_differentiable takes (media_tensor, prompts, records) and forwards records to the backend; update the recipe trainer and the SD3 refl callers to the new signature - move Trainer base from unirl/trainer/trainer.py to recipes/common/trainer.py - configs: reward _target_ -> RewardService; fix the placement comment (cross-process autograd is supported via GradContext RPC)

# Conflicts: # unirl/reward/service.py

…oles DSL - Rebase onto current main (Tencent-Hunyuan#214): RolloutReq/RolloutInputs are gone; the actor role now consumes Texts/Images primitives plus per-sample metadata records straight from the data-source Sample. - Drop recipes/common/ (role-list orchestration): REFLTrainer subclasses BaseTrainer directly and wires actor + reward with placement()+remote_hydra, mirroring RewardBackpropTrainer (the SD3 image-ReFL driver). ReflActorRole mirrors ReFLPolicy's family-agnostic contract (pipeline_target + model_config + from_config; FSDPBackend composed in initialize()). - Re-root both configs from the roles: list to the repo-wide flat schema (actor:/reward:/data_source:/sampling:/logging:), the same shape as examples/diffusion/refl_sd3.yaml. - KL correctness across DP shards: diffuse_with_grad now returns per-sample [B] KL (concat field) instead of a per-shard scalar shared field, so DP_SCATTER merge/re-shard round-trips each shard's own KL. Previously the driver collapsed all shards to one value (loss/logging skew; gradient flow was unaffected because dKL/dkl is the constant kl_weight). - I2V condition assembly moves out of the role into Wan21/Wan22ReflPipeline.build_refl_conditions (mirrors each mainline pipeline's generate); negative prompts ride sampler_kwargs. - Seed scheme now matches ReFLPolicy (base + 1000*rollout_id + dp_rank); previously every rollout redrew the same init noise. - reward/service.py: restore the List typing import lost in the merge.

Regression verification for the P1 review finding: the original per-shard scalar shared_field KL collapsed to rank 0 on DP collect and only lined up on the batch_size == actor_dp == 8 topology. kl_loss is now a per-sample [B] concat column; scripts/verify_refl_kl_batching.py pins chunk/cat round-trips, rewards/KL payload lockstep, per-shard backward grad shapes, unequal actor/reward dp re-chunking, and numeric equivalence with the legacy scalar mean at the pytree wire layer (CPU, no Ray). Shipped as a standalone runnable script rather than a tests/ tree per the Tencent-Hunyuan#99/Tencent-Hunyuan#267 no-unenforced-test-suite policy.

…dtype assert fp32 LoRA params over the bf16 base fail torch 2.7 FSDP2's uniform-original-dtype assertion at first forward (documented in examples/diffusion/refl_sd3.yaml). Omit master_dtype so LoRA stays bf16, matching the SD3 refl recipe's portable choice. Caught by the 8xH20 fleet smoke; the contributor's environment tolerated the fp32 mix.

…rs 5.6 stack Validated on 8xH20 in an isolated transformers==5.6.2 + peft==0.20.0 venv (real VideoReward checkpoint load + differentiable forward/backward: rewards finite, grad_abs_mean=2.2e-3): - vision features: prefer pooler_output — on 5.6 it holds the merged features that fill media placeholders; last_hidden_state is the PRE-merger states (4x tokens, vision dim) and silently corrupts the reward if scattered. Raw-Tensor branch kept for the 4.56 fleet image (TODO drop when the image moves to the locked stack); all other fallbacks removed per the pin-one-version policy. - checkpoint loader: apply the old→new Qwen2VL layout remap in the LoRA-split branch too (module paths are embedded in LoRA keys); guard against double-prefixing already-new-layout keys. - factory: stop forwarding use_cache through from_pretrained — 5.x passes unknown kwargs to the model ctor (TypeError); set it on the config after load, as the author's own disabled-shim note prescribed. - slim stale multi-era compat comments down to the load-bearing facts.

haonan3 · 2026-07-30T12:23:49Z

transformers 5.6 validation round at 811be0f9 — the declared core stack (>=5.6,<5.7) is not what the fleet image ships (4.56.0.dev0), so we validated in an isolated transformers==5.6.2 + peft==0.20.0 venv on 8xH20, with the real VideoReward checkpoint. Three 5.6 breaks found and fixed:

Vision features (reward_model.py): 5.6's vision tower returns BaseModelOutputWithPooling where pooler_output = merged features (what get_video_features reads) and last_hidden_state = PRE-merger states — the old preference for last_hidden_state would silently scatter wrong features into the placeholders. Now pooler_output wins; remaining fallbacks removed (pin-one-version policy), raw-Tensor branch kept only for the 4.56 fleet image with a removal TODO.
Checkpoint loader (checkpoint.py): the old→new Qwen2VL layout remap now also applies to the LoRA-split branch (LoRA keys embed module paths). Note: exercised statically only — no LoRA-split VideoAlign checkpoint was available; the full-model.pth branch is what the gate ran.
use_cache forwarding (factory.py): 5.x forwards unknown from_pretrained kwargs to the model ctor → TypeError; it is now set on the config after load (the fix the file's own disabled-shim note prescribed).

Gate evidence (isolated venv, GPU): VideoReward load → differentiable forward → backward — rewards=[-3.6562, -3.3438] (random-noise input), grad_abs_mean=2.2e-3, all finite. Env findings for the record: the fleet image carries transformers 4.56.0.dev0 / peft 0.17.1 (incompatible with 5.6) and lacks diffusers — the declared 5.6 stack currently exists in no image; and peft>=0.14 admits combinations that cannot import against transformers 5.6. A lockfile-driven image rebuild proposal will follow separately.

…ed transformers 5.6 only Per review direction: no version-compat branches; a wrong environment fails loudly and the user aligns the env, not the code. - _as_tensor: pooler_output only (drop the 4.56 raw-Tensor concession) - visual property: single 5.6 location (self.model.visual) - checkpoint remap: always applied (target layout is 5.6 by policy), idempotent per key - processor: rely on 5.6 fast-by-default; drop the force-swap + use_fast - attention: hardcode sdpa — flash-attn is not part of the locked stack - pyproject peft floor 0.14 → 0.20: older peft imports transformers cache symbols removed in 5.x and fails at import against the 5.6 pin Gate on 8xH20 (isolated transformers==5.6.2 + peft==0.20.0 venv, real VideoReward ckpt): load + differentiable forward/backward PASS, rewards=[-3.5938, -3.375], grad_abs_mean=3.5e-3.

…d README Review decisions: - recipes/refl/models/ → recipes/refl/model_adaptor/ — the directory holds algorithm-side adaptations wrapping core models to the recipe's BPTT contract, not model definitions; the old name collided with unirl/models semantics and blurred the graduation boundary. - Launch .sh wrappers deleted: zero logic beyond env-var placeholders; the launch surface is one documented command. recipes/refl/README.md now carries the launch examples, layout map, environment policy, and the verification table. Static at this head: compose x2 rc=0; 2270 _target_ paths resolve; KL topology checks pass; full pre-commit green.

into feat/wan-refl-recipes-v2

…ental/ Core is not just components — the official training flows live in core (train_*.py + the trainers). A layer named 'recipes' reads as the official, recommended usage and pulls that attention to the non-official tier; 'recipe' also already means the flat YAML in this repo's vocabulary (examples/ is the former recipes/ tree). experimental/ names what this tier actually is: the incubation side of a two-way flow — packages start here, mainstream ones get absorbed and solidified into core, ill-fitting core paths move down or out. Internal/private packages live here uncommitted. All dotpaths, the target checker, and docs updated; upstream provenance references (mmrl/recipes/...) intentionally untouched.

…mples/, reward/ Naming rule for the experimental tier (two-way flow): directories holding the SAME kind of content as a core location take the core name, so promotion/demotion is a structural no-op — refl/configs/ → refl/examples/ (mirrors the top-level examples/; graduates into examples/refl/) and refl/rewards/ → refl/reward/ (mirrors unirl/reward/; graduates into unirl/reward/local/). Different-kind content keeps a distinct name on purpose (model_adaptor/ wraps core models to the BPTT contract — it is not a models/ and graduates by merging, not by moving).

…r/ → models/ The mirror rule settles on one criterion: a directory is named after the core home its content graduates into. wan21.py / wan22.py define pipeline and stage classes — the same kind of artifact unirl/models/<m>/ holds — and graduate by merging into those model packages, so the directory mirrors unirl/models/ (plural, matching core exactly; reward/ is singular because unirl/reward/ is). The earlier model_adaptor name predates the mirror rule; the fork-confusion concern it addressed is now handled by the graduation notes in the layout table and module docstrings.

…nc main Mechanics PASS on the declared 5.6 stack (9.5h, 228s/step, clean exit, no reward collapse, grads healthy). Reward is FLAT at this horizon on substitute assets (pickscore prompts + VideoReward-2B): segment means 1.91/1.95/1.91/2.07/1.80/1.95, first25→last25 +2.4%, OLS slope ~0. Learning-effect verdict deferred to longer horizons / original assets; lr-sensitivity diagnostic tracked in PR Tencent-Hunyuan#210.

… table The 150-rollout run was a maintainer-side sanity diagnostic on substitute assets; its record lives in the PR Tencent-Hunyuan#210 discussion, not in the package's verification table, which lists only runs with product-level standing.

…mantics Checking pass on the flat trend run found this adjustment-era divergence: the role was rewriting params.seed per rollout/rank (decorrelated noise), while the contributor's verified 835-rollout curve trains DRaFT on a fixed initial noise (seed used verbatim, eta=0 ODE). Ship his regime; varying noise is a semantics change that needs its own evidence. Drops the now consumer-less rollout_id plumbing from generate_samples/train_step.

…es lora_A Checking follow-up on the flat trend run. The contributor's configs carried master_dtype: fp32; the adjustment dropped it to dodge FSDP2's uniform-original-dtype assert on the fleet image's torch 2.7.1. That assert is an environment-misalignment symptom, not a code constraint: the pinned torch family (2.11+, via the sglang extra) checks dtype uniformity over trainable params only, and the FSDP backend's master_dtype path implements exactly this bf16-base + fp32-LoRA-master regime. Checkpoint forensics on the 150-rollout run show the bf16 master froze lora_A entirely (delta ~0.0001% over 100 steps; AdamW step 5e-6 < bf16 ULP at |A|~0.1) while the contributor's fp32 runs train the full (A, B). Align the environment, not the config.

Same correction as the wan configs on #210: the bf16-master variant was an environment workaround for the fleet image's torch 2.7.1 (FSDP2 uniform-dtype assert over all params); the pinned torch checks trainables only, and the historical SD3 ReFL curves were produced with the fp32 master.

…vention Writes down the experimental-tier rules converged in the #210 review and makes the mechanical ones lint-enforced: - experimental/README.md — the contract: two-way flow with core (official flows live in core; mainstream packages graduate up, ill-fitting core paths move down), package anatomy with mirror naming (models/ reward/ examples/ named after the core home content graduates into), locked single-stack policy, additive-only requirements with the differentiable⇒same-process⇒same-stack corollary, owner+verification table requirement. - scripts/check_experimental_boundaries.py (pre-commit): core never imports experimental; packages never import each other; recipe requirements may not re-declare core dependencies (same-process colocation cannot version-isolate). - experimental/private_*/ gitignored; the tier is not packaged into the wheel, so private code cannot leak into a build.

The index gate hardcoded PACKAGE = "unirl" from before main broadened _TARGET_RE to unirl|experimental (#210), so experimental.* targets were extracted but could never resolve. Derive the regex from PACKAGES so the accepted roots and the index can't drift apart again.

scripts/ kept accumulating non-guard files because its name promised generic tooling space (#158's ep_verify/, #210's verify script). Name the folder after its real contract instead: lint/ holds exactly the scripts wired into .pre-commit-config.yaml, and CLAUDE.md now states the positive rule (verification harness results are quoted in the PR Test Plan, not committed).

github-actions Bot added the need review Ready and waiting for review label Jul 15, 2026

YSunLIN changed the title ~~Decouple training algorithms from UniRL Core~~ feat(recipes): Decouple training algorithms from UniRL Core Jul 15, 2026

haonan3 requested review from CjhHa1, celve and haonan3 and removed request for celve July 16, 2026 12:04

haonan3 requested review from leviking98z-rgb and removed request for CjhHa1 July 16, 2026 12:58

CjhHa1 mentioned this pull request Jul 26, 2026

fix(config): make the cross-component recipe contracts actually gate #257

Draft

2 tasks

yohunawu and others added 8 commits July 27, 2026 16:28

new feature: support wan series refl recipes.

4b552f0

rename wan2.2 i2v refl config file's name

db35f4c

fix bug: resolved the issue of being unable to enter the kl branch in…

c1333d5

… the wan refl recipe.

fix bug: support batched WAN VAE tiled decode instead of fixing the b…

ced3c37

…atch size to 1

refactor: move base role from unirl core to recipes

25d4ee7

refactor: move reward role class to recipes/common/roles.py

dbc4889

YSunLIN force-pushed the feat/wan-refl-recipes branch from de650ea to ec87340 Compare July 27, 2026 13:47

Merge main and resolve ReFL recipe compatibility

e3c6b94

YSunLIN force-pushed the feat/wan-refl-recipes branch from ec87340 to e3c6b94 Compare July 27, 2026 13:53

haonan3 added 4 commits July 29, 2026 17:47

Merge remote-tracking branch 'origin/main' into feat/wan-refl-recipes-v2

dbcdd19

# Conflicts: # unirl/reward/service.py

Merge remote-tracking branch 'origin/main' into feat/wan-refl-recipes-v2

171d26f

haonan3 requested review from KemingWu, Zcchill, celve and xshrz as code owners July 30, 2026 07:31

haonan3 added 2 commits July 30, 2026 18:14

haonan3 added 5 commits July 30, 2026 21:00

Merge branch 'main' into feat/wan-refl-recipes

442b9aa

Merge branch 'feat/wan-refl-recipes' of https://github.com/YSunLIN/UniRL

27bd2ec

into feat/wan-refl-recipes-v2

haonan3 changed the title ~~feat(recipes): add WAN ReFL as a self-contained recipe~~ feat(experimental): add WAN ReFL as a self-contained training package Jul 30, 2026

haonan3 added 2 commits July 30, 2026 22:07

haonan3 added 5 commits July 31, 2026 06:51

Merge remote-tracking branch 'origin/main' into feat/wan-refl-recipes-v2

8401d22

haonan3 approved these changes Jul 31, 2026

View reviewed changes

haonan3 merged commit 411901e into Tencent-Hunyuan:main Jul 31, 2026
5 checks passed

haonan3 mentioned this pull request Jul 31, 2026

chore(scripts): drop ad-hoc verify harnesses, move dataset converters to datasets/ #284

Merged

2 tasks

leviking98z-rgb mentioned this pull request Jul 31, 2026

[RFC] Refactor and simplify the video track #285

Open

This was referenced Jul 31, 2026

perf(lint): make the recipe _target_ guard filesystem-cheap #283

Merged

fix: post-merge review sweep — lint blind spots, bagel prefill guard, label workflows, refl nits #290

Merged

Navigation Menu

Uh oh!

feat(experimental): add WAN ReFL as a self-contained training package - #210

feat(experimental): add WAN ReFL as a self-contained training package#210
haonan3 merged 29 commits into
Tencent-Hunyuan:mainfrom
YSunLIN:feat/wan-refl-recipes

YSunLIN commented Jul 15, 2026 •

edited by haonan3

Loading

Uh oh!

haonan3 commented Jul 16, 2026

Uh oh!

celve commented Jul 22, 2026

Uh oh!

haonan3 commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

YSunLIN commented Jul 15, 2026 • edited by haonan3 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

feat(experimental): add WAN ReFL as a self-contained training package

Summary

Review adjustments (2026-07-30, maintainer push at ec561c7a)

Test Plan

Verification Status

Wan 2.1 T2V ReFL training reward curve

Wan 2.2 I2V ReFL training reward curve

Compatibility / Risk

Uh oh!

haonan3 commented Jul 16, 2026

Uh oh!

celve commented Jul 22, 2026

Uh oh!

haonan3 commented Jul 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

YSunLIN commented Jul 15, 2026 •

edited by haonan3

Loading

Review adjustments (2026-07-30, maintainer push at `ec561c7a`)